Transition Matrix

1 Ways to Describe a Stochastic Process We will use the biased coin example to illustrate three ways with which we can describe a stochastic process, a Markov Chain (MC) in particular. Probabilistic Random variable X represents the outcome of the experiment where we toss a specific coin once. There is 0.55 probability for the result to be heads (H) and 0.45 to be tails (T). X i = H with probability 0.55 T with probability 0.45 Random Walk Figure 1a shows a topological description of a sequence of coin tossings. Figure 1b illustrates a process which is relatively straight forward to define as a graph, but rather complicated to describe probabilistically. (a) Biased Coin. (b) It is not trivial to define the random variables which describe this system. A matrix description of the random walks 1a and 1b is P a and P b respectively. Variable x in P b denotes a non-zero value. For example, P b (2, 2) is the probability of state 2 to remain in this state and P b (2, 3) is the probability of moving to state 3. H T P a = 0.55 0.45 0.55 0.45 H T P b =   0 x 0 0 x x x 0 0   2 Strongly Connected Components In the past lecture we mentioned different types of Markov states: absorbing, transient, recurrent, periodic. We will now introduce them in terms of strongly 1